Hessian Free Deep Learning

Optimization techniques used in Machine Learning play an important role in the training of the Neural Network in regression and classification tasks. Predominantly, first order optimization methods such as Gradient Descent have been used in the training of Neural Networks, since second order methods, such as Newton’s method, are computationally infeasible. However, second order methods show much better convergence characteristics than first order methods, because they also take into account the curvature of the error space. Additionally, first order methods require a lot of tuning of the decrease parameter, which is application specific. They also have a tendency to get trapped in local optimum and exhibit slow convergence. Thus Newton’s method is absolutely essential to train networks with deep architectures. The reason for in-feasibility of Newton’s method is the computation of the Hessian matrix, which takes prohibitively long. Influential work by Pearlmutter [2] led to development of a method of using the Hessian without actually computing it. Recent work [1] has involved training of a deep network consisting of a number of Restricted Boltzmann Machine using Newton’s method without directly computing the Hessian matrix, in a form of “Hessian free” learning. The method had exhibited success on the MNIST handwriting recognition data set when used to train an Restricted Boltzmann Machine using Hinton’s [3] method, with a better quality solution for classification tasks. The proposed work for the CS229 project aims to improve upon the method of “Hessian Free” (HF) learning and apply it to different classification tasks. To do this, the Hessian free learning method will be implemented and results for the experiments using MNIST will be replicated. Through analysis, it is aimed to propose further modifications that will improve the method and also run it on different classification tasks.